deLearyous: Training Interpersonal Communication Skills Using Unconstrained Text Input

We describe project deLearyous, in which the goal is to develop a proof-of-concept of a serious game that will assist in the training of communication skills following the Interpersonal Circumplex (also known as Leary’s Rose) –a framework for interpersonal communication. Users will interact with the application using unconstrained written natural language input and will engage in conversation with a 3D virtual agent. The application will thus alleviate the need for expensive communication coaching and will offer players a non-threatening environment in which to practice their communication skills. We outline the preliminary data collection procedure, as well as the workings of each of the modules that make up the application pipeline. We evaluate the modules’ performance and offer our thoughts on what can be expected from the final “proof-of-concept” application. To get a firm grasp on the structure and dynamics of human-to-human conversations, we first gathered data from a series of “Wizard of Oz” experiments in which the virtual agent was replaced with a human actor. All data was subsequently transcribed, analysed and annotated. This data functioned as the basis for all modules in the application pipeline: the NLP module, the scenario engine, the visualization module, and the audio module. The freeform, unconstrained text input from the player is first processed by a Natural Language Processing (NLP) module, which uses machine learning to automatically identify the position of the player on the Interpersonal Circumplex. The NLP module also identifies the topic of the player’s input using a keyword-based approach. The output of the NLP module is sent to the scenario engine, which implements the virtual agent’s conversation options as a finite state machine. Given the virtual agent’s previous state and Circumplex position, it predicts the most likely follow-up state. The follow-up state is then realized by the visualization and audio modules. The visualization module takes care of displaying the 3D virtual agent’s facial and torso animations, while the audio module looks up and plays the appropriate pre-recorded audio responses. In terms of performance, the NLP module appears to be a bottleneck, as finding the position of the player on the Interpersonal Circumplex is a very difficult problem to solve automatically. However, we show that human agreement on this task is also very low, indicating that there isn’t always a single “correct” way to interpret Circumplex positions. We conclude by stating that applications like deLearyous show promise, but we also readily admit that technology still has a way to go before they can be used without human supervision.